This is an R Markdown Notebook
Your project - Detect Anomaly in Industrial Process with Deep Learning
Deep Learning… In this lecture I would like to esplore with you the DeepLearning we can do with H2O package in R…
Data Exploration
library(plotly)
Attaching package: 㤼㸱plotly㤼㸲
The following object is masked from 㤼㸱package:ggplot2㤼㸲:
last_plot
The following object is masked from 㤼㸱package:stats㤼㸲:
filter
The following object is masked from 㤼㸱package:graphics㤼㸲:
layout
About the NORMAL dataset
This dataset was selected .. after .. the period of deep process maintenance
plot_ly(z = NORM_2016, type = "surface")
About the TEST dataset
This dataset contains some values that may indicate potential anomaly
About the ANOMALY dataset
This dataset contains some values that indicate the anomaly
Summary: 3 dataset were selected!
Train Deep Learning Model
normality_model <- h2o.deeplearning(x = names(train),
training_frame = train,
activation = "Tanh",
autoencoder = TRUE,
hidden = c(8,5,8),
sparse = TRUE,
l1 = 1e-4,
epochs = 100)
|
| | 0%
|
|============================================= | 40%
|
|========================================================================================== | 80%
|
|=================================================================================================================| 100%
Check MSE

shutdown JVM
h2o.shutdown(prompt = F)
[1] TRUE
Homework:
- try different activation function, each time calculate MSE
- try sparse false parameter, what is changing?
- try high and veryhigh complexity of the model, calculate MSE, conclude which one is the best
- try 10:100:10 is it work?
- try 200:200 will it work? what is the resulting MSE
- try 5 hidden layers
- try replicate_training data = False, how much time is different?
- weights_column this is the observation weight, try to add 1 column with observation weights of importance! see help
LS0tDQp0aXRsZTogIkNyZWF0aW5nIEFub21hbHkgRGV0ZWN0aW9uIFN5c3RlbSBmb3IgSW5kdXN0cmlhbCBQcm9jZXNzIHdpdGggRGVlcCBMZWFybmluZyINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDogZGVmYXVsdA0KICBodG1sX25vdGVib29rOiBkZWZhdWx0DQogIHBkZl9kb2N1bWVudDogZGVmYXVsdA0KLS0tDQoNClRoaXMgaXMgYW4gW1IgTWFya2Rvd25dKGh0dHA6Ly9ybWFya2Rvd24ucnN0dWRpby5jb20pIE5vdGVib29rDQoNCg0KIyMjIFlvdXIgcHJvamVjdCAtIERldGVjdCBBbm9tYWx5IGluIEluZHVzdHJpYWwgUHJvY2VzcyB3aXRoIERlZXAgTGVhcm5pbmcNCg0KRGVlcCBMZWFybmluZy4uLiBJbiB0aGlzIGxlY3R1cmUgSSB3b3VsZCBsaWtlIHRvIGVzcGxvcmUgd2l0aCB5b3UgdGhlIERlZXBMZWFybmluZyB3ZSBjYW4gZG8gd2l0aCBIMk8gcGFja2FnZSBpbiBSLi4uDQoNCiMjIyBEYXRhIEV4cGxvcmF0aW9uDQoNCmBgYHtyfQ0KIyBsaWJyYXJpZXMNCmxpYnJhcnkodGlkeXZlcnNlKQ0KbGlicmFyeShwbG90bHkpDQoNCiMgZGF0YSByZWFkaW5nDQpERl9OT1JNQUwgPC0gcmVhZF9yZHMoIkRBVEEtbm9ybWFsLnJkcyIpDQpERl9URVNUIDwtIHJlYWRfcmRzKCJEQVRBLXRlc3QucmRzIikNCkRGX0FOT01BTFkgPC0gcmVhZF9yZHMoIkRBVEEtYW5vbWFseS5yZHMiKQ0KDQojIG5hbWVzIG9mIHRoZSBkYXRhc2V0IGNvbHVtbnMNCm5hbWVzKERGX05PUk1BTCkNCmBgYA0KDQojIyMgQWJvdXQgdGhlIE5PUk1BTCBkYXRhc2V0DQoNCmBgYHtyfQ0KTk9STV8yMDE2IDwtIERGX05PUk1BTCAlPiUgc2VsZWN0KDI6MTEpICU+JSBhcy5tYXRyaXgoKSANCnN1bW1hcnkoTk9STV8yMDE2KQ0KDQpgYGANCg0KVGhpcyBkYXRhc2V0IHdhcyBzZWxlY3RlZCAuLiBhZnRlciAuLiB0aGUgcGVyaW9kIG9mIGRlZXAgcHJvY2VzcyBtYWludGVuYW5jZQ0KDQpgYGB7cn0NCnBsb3RfbHkoeiA9IE5PUk1fMjAxNiwgdHlwZSA9ICJzdXJmYWNlIikNCmBgYA0KDQojIyMgQWJvdXQgdGhlIFRFU1QgZGF0YXNldA0KDQpUaGlzIGRhdGFzZXQgY29udGFpbnMgc29tZSB2YWx1ZXMgdGhhdCBtYXkgaW5kaWNhdGUgcG90ZW50aWFsIGFub21hbHkNCg0KYGBge3J9DQpURVNUXzIwMTUgPC0gREZfVEVTVCAlPiUgc2VsZWN0KDI6MTEpICU+JSBhcy5tYXRyaXgoKQ0KcGxvdF9seSh6ID0gVEVTVF8yMDE1LCB0eXBlID0gInN1cmZhY2UiKQ0KDQpgYGANCg0KIyMjIEFib3V0IHRoZSBBTk9NQUxZIGRhdGFzZXQNCg0KVGhpcyBkYXRhc2V0IGNvbnRhaW5zIHNvbWUgdmFsdWVzIHRoYXQgaW5kaWNhdGUgdGhlIGFub21hbHkNCg0KYGBge3J9DQoNClRFU1RfMjAxNyA8LSBERl9BTk9NQUxZICU+JSBzZWxlY3QoMjoxMSkgJT4lIGFzLm1hdHJpeCgpDQpwbG90X2x5KHogPSBURVNUXzIwMTcsIHR5cGUgPSAic3VyZmFjZSIpDQoNCmBgYA0KDQpTdW1tYXJ5OiAzIGRhdGFzZXQgd2VyZSBzZWxlY3RlZCENCg0KDQoNCg0KDQoNCg0KDQojIyMjIFRyYWluIERlZXAgTGVhcm5pbmcgTW9kZWwNCg0KDQoNCg0KDQpgYGB7cn0NCmxpYnJhcnkoaDJvKQ0KaDJvLmluaXQobnRocmVhZHMgPSAyKQ0KDQp0cmFpbiA8LSBhcy5oMm8oeCA9IE5PUk1fMjAxNiwgZGVzdGluYXRpb25fZnJhbWUgPSAidHJhaW4iKQ0KdGVzdCA8LSBhcy5oMm8oeCA9IFRFU1RfMjAxNSwgZGVzdGluYXRpb25fZnJhbWUgPSAidGVzdCIpDQphbm9tYWx5IDwtIGFzLmgybyh4ID0gVEVTVF8yMDE3LCBkZXN0aW5hdGlvbl9mcmFtZSA9ICJhbm9tYWx5IikNCg0KIyA/aDJvLmRlZXBsZWFybmluZw0KDQpub3JtYWxpdHlfbW9kZWwgPC0gaDJvLmRlZXBsZWFybmluZyh4ID0gbmFtZXModHJhaW4pLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0cmFpbmluZ19mcmFtZSA9IHRyYWluLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhY3RpdmF0aW9uID0gIlRhbmgiLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhdXRvZW5jb2RlciA9IFRSVUUsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhpZGRlbiA9IGMoOCw1LDgpLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzcGFyc2UgPSBUUlVFLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGwxID0gMWUtNCwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXBvY2hzID0gMTAwKQ0KDQoNCmBgYA0KDQojIyMjIHJlY29uc3RydWN0IGRhdGFzZXQNCg0KYGBge3J9DQojIHJlY3JlYXRlIA0KdGVzdF9yZWNvbiA8LSBoMm8ucHJlZGljdChub3JtYWxpdHlfbW9kZWwsIHRyYWluKSAlPiUgYXMubWF0cml4KCkNCnBsb3RfbHkoeiA9IHRlc3RfcmVjb24sIHR5cGUgPSAic3VyZmFjZSIpDQpgYGANCg0KIyMjIyBDaGVjayBNU0UNCg0KDQpgYGB7cn0NCg0KbXNlIDwtIGgyby5hbm9tYWx5KG5vcm1hbGl0eV9tb2RlbCwgdHJhaW4pICU+JSBhcy5kYXRhLmZyYW1lKCkNCm1zZV90ZXN0IDwtIGgyby5hbm9tYWx5KG5vcm1hbGl0eV9tb2RlbCwgdGVzdCkgJT4lIGFzLmRhdGEuZnJhbWUoKQ0KbXNlX2Fub20gPC0gaDJvLmFub21hbHkobm9ybWFsaXR5X21vZGVsLCBhbm9tYWx5KSAlPiUgYXMuZGF0YS5mcmFtZSgpDQoNCm1zZSRsYWJlbCA8LSAibm9ybWFsIg0KbXNlX3Rlc3QkbGFiZWwgPC0gInRlc3QiDQptc2Vqb2luZWQgPC0gcmJpbmQobXNlLCBtc2VfdGVzdCkNCm1zZWpvaW5lZCRpbmRleCA8LSAxOm5yb3cobXNlam9pbmVkKQ0KDQpnZ3Bsb3QobXNlam9pbmVkLCBhZXMoeCA9IGluZGV4LCB5ID0gUmVjb25zdHJ1Y3Rpb24uTVNFLCBjb2wgPSBhcy5mYWN0b3IobGFiZWwpKSkgKyBnZW9tX2xpbmUoKQ0KDQpwbG90LnRzKG1zZSkNCnBsb3QudHMobXNlX3Rlc3QpDQpwbG90LnRzKG1zZV9hbm9tKQ0KDQoNCm1zZV9hbm9tJGxhYmVsIDwtICJhbm9tYWx5Ig0KbXNlX2FsbCA8LSByYmluZChtc2UsbXNlX3Rlc3QsIG1zZV9hbm9tKQ0KbXNlX2FsbCRpbmRleCA8LSAxOm5yb3cobXNlX2FsbCkNCmdncGxvdChtc2VfYWxsLCBhZXMoeCA9IGluZGV4LCB5ID0gUmVjb25zdHJ1Y3Rpb24uTVNFLCBjb2wgPSBhcy5mYWN0b3IobGFiZWwpKSkgKyBnZW9tX2xpbmUoKQ0KDQpgYGANCg0KIyBzaHV0ZG93biBKVk0NCmBgYHtyfQ0KaDJvLnNodXRkb3duKHByb21wdCA9IEYpDQpgYGANCg0KDQpIb21ld29yazoNCg0KLSB0cnkgZGlmZmVyZW50IGFjdGl2YXRpb24gZnVuY3Rpb24sIGVhY2ggdGltZSBjYWxjdWxhdGUgTVNFDQotIHRyeSBzcGFyc2UgZmFsc2UgcGFyYW1ldGVyLCB3aGF0IGlzIGNoYW5naW5nPw0KLSB0cnkgaGlnaCBhbmQgdmVyeWhpZ2ggY29tcGxleGl0eSBvZiB0aGUgbW9kZWwsIGNhbGN1bGF0ZSBNU0UsIGNvbmNsdWRlIHdoaWNoIG9uZSBpcyB0aGUgYmVzdA0KLSB0cnkgMTA6MTAwOjEwIGlzIGl0IHdvcms/DQotIHRyeSAyMDA6MjAwIHdpbGwgaXQgd29yaz8gd2hhdCBpcyB0aGUgcmVzdWx0aW5nIE1TRQ0KLSB0cnkgNSBoaWRkZW4gbGF5ZXJzDQotIHRyeSByZXBsaWNhdGVfdHJhaW5pbmcgZGF0YSA9IEZhbHNlLCBob3cgbXVjaCB0aW1lIGlzIGRpZmZlcmVudD8NCi0gd2VpZ2h0c19jb2x1bW4gdGhpcyBpcyB0aGUgb2JzZXJ2YXRpb24gd2VpZ2h0LCB0cnkgdG8gYWRkIDEgY29sdW1uIHdpdGggb2JzZXJ2YXRpb24gd2VpZ2h0cyBvZiBpbXBvcnRhbmNlISBzZWUgaGVscA0K